Expiry Date Monitor
ECE 5725 Fall 2019
Melaney Chen (yc774) & Yeolim Jo (yj248)
Demonstration Video
Introduction
There are two aspects to this project. The first aspect is for foods with set expiration dates. The RPi prompts the user to take a picture of the food item, shows the picture, and gives the user an option to retake the picture. Once the user is happy with the picture, they then take a picture of the expiration date and are again given the option to retake the picture until it shows up clear on the screen. Then the RPi performs optical character recognition (OCR) on the expiration date picture, and asks the user to confirm whether the recognized date is correct. If not, the screen switches to a number touchpad, and the user is able to enter the expiration date. Once the expiration date passes, an email is sent to the user containing a picture of the item, and tells them that this food item has expired.
The second aspect is for foods that should be consumed within a specified time period after opening (i.e. canned items after a week of opening). A container with integrated hardware (our “can-tainer”!) stores opened cans, and a built-in distance sensor detects if the can has been opened recently, sending you a reminder to eat your food if untouched for a few days, and after a week since the can was first opened, sends you an email that the food has expired. There is a button on the can that allows you to reset the timer when a new food has been placed in the can-tainer.
Project Objective:
- Use computer vision and optical character recognition to read and interpret product expiration dates
- Have a sensor that communicates remotely with the RPi
- Have the RPi store and track food expiration dates and send email reminders whenever food has expired
Design
PiTFT-pygame interface
The main program on the RPi was created using pygame. The four buttons on the side of the PiTFT were used to advance through the steps of taking pictures and storing the expiration date. The four button options were “Start”, “Take Picture”, “Yes”, and “No”. The buttons were implemented using callbacks. Many global variables that tracked the states were initialized such that even though the four buttons could be pressed at any time, they would only do things if we were in the correct state (for instance, the “take picture” button would not take a picture unless there was a combination of states that had the global variable “can_take_picture” set to True.)
Simplified flowchart summary of picture-taking interface on the PiTFT
RPi camera picture-taking and OCR process
We hooked up the PiCamera to the RPi and installed the OCR package using sudo apt-get install tesseract-ocr
The hardware component of this was really simple - our RPi and PiTFT were already in the case from the labs. There was velcro on the bottom of the case, and we put some velcro on the PiCamera holder and attached it to the back of the RPi case, and it looked like a camera!
Can opening sensing system
We wanted a simple system that was compact and could be easily packaged in our Can-tainer. We considered various sensors such as break-beam sensors, IR reflectance sensors, and distance sensors. We wanted a one-sided sensor so that only either the lid or the body of the Can-tainer would need electronics integrated. This ruled out break beam sensors since this required having the transmitter on the body and the receiver on the lid. The IR reflectance sensor did not arrive on time, so we settled on a distance sensor.
This sensor would maintain an output HIGH signal, and turn LOW when an object within 5cm away is detected. Therefore, we expect the sensor output to stay LOW whenever a can is detected inside the Can-tainer, and rise to HIGH when the lid is removed.
We also needed a way to transmit the signal from the sensor to the RPi itself, so we considered untethered forms of communication: bluetooth, WiFi, RF. We decided on WiFi since the RPi was already configured to work with WiFi and because Melaney had some experience working with WiFi-enabled microcontrollers for her MEng project. We chose to use an Adafruit Feather ESP32 board since it is relatively small so we could package it well, and because we had some readily available.
The final design: 1x Adafruit HUZZAH32 – ESP32 Feather Board, 1x Pololu Carrier with Sharp GP2Y0D805Z0F Digital Distance Sensor 5cm
Can-tainer lid: wiring side and sensing side
Can-tainer (3D printed)
Our Can-tainer would need to be able to hold a standard 16oz can, so we designed our container to accommodate those dimensions.
In SolidWorks, we were able to model the threads easily and confirmed with RPL that they could be printed, so we decided on a twist cap closure method. This would also provide a secure closure without any additional latch hardware.
We wanted to keep the electronics hidden away and secured, so we designed a compartment in the container lid that would house the distance sensor, button, ESP32 board, and any accompanying wiring.
The top of the lid had a small hole so a button could fit through. The bottom of the lid had slots so the distance sensor could look through down into the main container body to detect for a can.
Can-tainer CAD
Can-tainer CAD: exploded view
Expiry monitor logic
We have a python file that simultaneously checks 3 files
- One contains the expiry date and file path to the image of the corresponding food from the RPi
- One contains the time at which the Can-tainer first received a newly opened can of food
- One contains the time at which the Can-tainer was last opened
When a photo is taken, the expiry date and the file path to the image is appended to the .txt file. When the reset button is pressed, the text file containing first opened date information is updated with the current time. When the can lid is opened (the sensor detects no object), the text file containing the last opened date is updated.
Testing
RPi - Testing the OCR
We first familiarized ourselves with the OCR software by using screenshots of computer print. Early on, we tried using the OCR on food wrappers and did not have much success. We used a wrapper whose expiration date was printed in dotted letters, and the OCR was not successful interpreting the picture. We then attempted to test using handwritten block letters. This was not very successful, and would often just return empty text. We implemented finding the contours of a picture and saving it in black and white with OpenCV to help strengthen the text. Still we had a lot of trouble with the software finding shadows and converting to contours. So then we used printed computer print. We initially had trouble because the camera could not take good close up pictures of the text because the focal point was off. We had mentioned this to Joe, and he pointed us to this little plastic tool that lets you manually focus the camera. We adjusted the focus and now could take better, close up pictures. A huge breakthrough was made when we realized that the brightness of the camera setting was incredibly important. We had a lot more success with a brighter camera setting, and the contour image processing was very successful at finding the black text on white paper. We did not go back and test with handwritten block letters, but instead decided to demo with a printed sheet of paper with a picture of food and a made up expiration date.
Experimenting with OCR using screenshots and various fonts
Testing the Feather ESP32 Microcontroller and wifi connection
The distance sensor would be connected to the ESP32, which would then need to transmit data over WiFi to the RPi. First we had to make sure that the WiFi communication between the two was possible. The ESP32 accepts code from the Arduino IDE, so we uploaded a simple sketch that would connect to the RPi, which in turn was running a Python script that would listen to any incoming clients. This worked successfully, so we continued with developing the code.
However, since the ESP32 requires a WiFi SSID and password to log into the WiFi, we could not use the available networks (Eduroam and RedRover) since these had more complicated login steps. To solve this, we used a hotspot connection from our iPhones, which both the ESP32 board and the RPi was connected to. At times we also connected our laptops to this hotspot so we could SSH into the RPi easily.
The overall function is such that each time the sensor detects the absence of a can in front of it, an interrupt in the code would start a WiFi client that connects to the RPi, which is listening for clients. The ESP32 sends a message that encodes which can was opened, so the Python script running on the RPi can update the correct corresponding .txt files.
Testing the distance sensor
We wired up the distance sensor to the ESP32 using a breadboard to test how to interpret the output signals of the sensor. We realized that using an interrupt that detects a RISING signal was best since we only wanted the WiFi communication function to be called once when the can is lifted.
The distance sensor worked very reliably so progress was smooth after that.
Flowchart summary of picture-taking interface on the PiTFT
Result
We managed to get a working system to demonstrate the function of our project within the time span of our demo. Instead of waiting for days for the food to expire, we set up our code so that the expiry email would be sent 30 seconds after taking the picture of the food or pressing the reset button, and the reminder email would be sent after 10 seconds of not opening the can lid. Our program sent emails as expected and attached the corresponding images successfully.
What went wrong
During testing and the demo, several issues occurred. We were able to solve most of them, so here is a summary of our troubleshooting:
- The email sending failed and returned an “unable to log on” error. This was because our Gmail account detected multiple sign-in attempts from the RPi and secured the account, locking the RPi device out. This was solved by logging into our Gmail account within the RPi from the browser, so that our Gmail account would recognize this device as a secure one
- Our script uses the datetime package to convert strings to a datetime object. Sometimes hidden characters in the .txt file such as ‘/n’ would be included in the string, and the datetime conversion would fail since the format was unrecognized. This was solved by using vim to read our .txt files to ensure they started out as 0 characters and 0 lines.
- The hotspot connection would work, but the ESP32 would not be able to communicate with the RPi server. We were unable to ping the ESP32 from the RPi, even though we could ping both from our laptop. The solution that fixed this was to change the iPhone that was providing the hotspot, although we do not understand why that solved the problem nor why the problem occurred in the first place.
- Power supply - when the charge of the lithium ion battery was low, the wifi connection would cut out. We fixed this by making sure the battery was fully charged.
Conclusion
The basic function of our project was successful, and we received reminder and expiry emails as intended (over shorter time spans of minutes instead of days). We scaled down our project, since we originally intended to have multiple cans, although adding more is easily scalable. The user input functions worked well: the button press on the can registered resets and the manual data entry buttons on the PiTFT screens were definitely a good addition to account for incorrect OCR readings. OCR worked well with clearly printed dates, and lighting was very important to the success of the character recognition. Shadows would really mess up the result. A lot of expiry labels are printed in dotted fonts, which definitely did not work.
Our hardware was packaged well and no electronics or wires were visible. We soldered our button and accompanying current limiting resistors to a small perf board, which fit comfortably in the electronics compartment of the Can-tainer lid. The Can-tainer itself was 3D printed to our desired specifications, so it was robust, functional, and clean-looking. The RPi, PiTFT and RPi camera came as one rigid unit after we used velcro to secure the RPi camera to the back of the RPi case. Overall, we were pleased with the overall result since our project worked successfully and this was a new experience for both of us as mechanical engineering students.
Future Work
- Developing OCR with actual labels
- Improved fixturing of electronics
- More physical can-tainers
- PiTFT screen could have a menu that has a ‘library’ function that allows you to browse through everything
- Implement deep sleep on the ESP32 to save power
- Erasing items off of the tracking text file after they expired
Work Distribution
Project group picture
Worked together on: can-tainer CAD, testing and debugging
Melaney Chen
yc774@cornell.edu
Yeolim Jo
yj248@cornell.edu
Parts List
- Raspberry Pi (provided in lab)
- PiTFT (provided in lab)
- Button (provided in lab)
- Raspberry Pi Camera V2 $25.00
- Polulu GP2Y0D805Z0F Digital Distance Sensor 5cm - $7.49
- Adafruit HUZZAH32 - ESP32 Feather Board - $19.95
Total: $52.44
References
PiCamera DocumentECE 5725 Fall 2018 Project: Automatic Number Plate Recognition System
ECE 5725 Spring 2019 Project: Laser 3D Scanner
Pololu Distance Sensor
Wifi with Python sockets
Open CV Contours
Code Appendix
Expiry Monitor on RPi
# ExpiryMonitor.py
#!/usr/bin/env python2
import select
import socket
import datetime
import time
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email import encoders
class RPiFood:
def __init__(self, Expiry,ImgPath):
self.expiry = datetime.datetime.strptime(Expiry, '%Y-%m-%d %H:%M:%S.%f')
self.imgPath = ImgPath
def SendExpiryEmail(ContainsImage,ImageFileName=' ',CanNumber = 0):
email_user = 'melchenna@gmail.com'
email_password = 'ece5725_2019'
email_send = 'yj248@cornell.edu'
subject = 'Your Food is About to Expire!'
msg = MIMEMultipart()
msg['From'] = email_user
msg['To'] = email_send
msg['Subject'] = subject
if ContainsImage:
body = 'Hi there, this is an email to let you know that this item expires today!'
msg.attach(MIMEText(body,'plain'))
# File attachment
attachment = open(ImageFileName,'rb')
FoodImage = MIMEBase('application','octet-stream')
FoodImage.set_payload((attachment).read())
encoders.encode_base64(FoodImage)
FoodImage.add_header('Content-Disposition',"attachment; filename= " + ImageFileName)
msg.attach(FoodImage)
else:
body = 'Hi there, this is an email to let you know that the item in Can #'+str(CanNumber)+' expires today!'
msg.attach(MIMEText(body,'plain'))
text = msg.as_string()
# Send it
server = smtplib.SMTP('smtp.gmail.com',587)
server.starttls()
server.login(email_user,email_password)
server.sendmail(email_user,email_send,text)
server.quit()
def SendReminderEmail(CanNumber):
email_user = 'melchenna@gmail.com'
email_password = 'ece5725_2019'
email_send = 'yj248@cornell.edu'
subject = 'Don\'t forget to eat your food!'
msg = MIMEMultipart()
msg['From'] = email_user
msg['To'] = email_send
msg['Subject'] = subject
body = 'Hi there, this is an email to let you know that you haven\'t opened Can #'+str(CanNumber)+' in a while!'
msg.attach(MIMEText(body,'plain'))
text = msg.as_string()
# Send it
server = smtplib.SMTP('smtp.gmail.com',587)
server.starttls()
server.login(email_user,email_password)
server.sendmail(email_user,email_send,text)
server.quit()
def CanFileToList(FileOutput):
num = len(FileOutput)
ParsedFile=[]
for i in range(0,num-1):
ParsedFile.append(FileOutput[i][:-1])
return ParsedFile
def ReadFile(FileName):
# Read info file and update
CanFile = open(FileName,"r+")
CanFileOutput = CanFile.readlines()
CanFile.close()
return CanFileOutput
def WriteFile(FileName,Record):
# Write record into file
CanFile = open(FileName,"w")
ListToWrite = []
for i in Record:
ListToWrite.append(str(i)+"\n")
ListToWrite.append("END")
CanFile.writelines(ListToWrite)
CanFile.close()
# For RPi
def RPiFileToList(FileOutput):
num = len(FileOutput)
ParsedFile=[]
for i in range(1,num/2+1):
Item = RPiFood(FileOutput[2*i-2][:-1],FileOutput[2*i-1])
ParsedFile.append(Item)
return ParsedFile
def CompareTime(FirstOpened,LastOpened):
deltaLastOpened = []
deltaFirstOpened = []
for i,j in zip(FirstOpened,LastOpened):
FirstOpenedDT = datetime.datetime.strptime(i, '%Y-%m-%d %H:%M:%S.%f')
LastOpenedDT = datetime.datetime.strptime(j, '%Y-%m-%d %H:%M:%S.%f')
TimeNow = datetime.datetime.now()
#LongestTimeUnopened = max((LastOpenedDT-FirstOpenedDT).total_seconds(),(TimeNow-LastOpenedDT).total_seconds())
LongestTimeUnopened = (TimeNow-LastOpenedDT).total_seconds()
TimeElapsedSinceOpen = (TimeNow-FirstOpenedDT).total_seconds()
deltaLastOpened.append(LongestTimeUnopened)
deltaFirstOpened.append(TimeElapsedSinceOpen)
#print(LongestTimeUnopened)
return deltaLastOpened, deltaFirstOpened
def SendReminder(DeltaLastOpenedSeconds,DeltaFromFirstOpenSeconds,RPiExpiry):
CanNumberActual = 0
#anSentRecently = ((datetime.datetime.now()-CanLastSent).total_seconds() < 10)
#RPiSentRecently = ((datetime.datetime.now()-RPiLastSent).total_seconds() < 10)
for i,j in zip(DeltaLastOpenedSeconds,DeltaFromFirstOpenSeconds):
CanNumberActual += 1
print(i)
if (i >= 10) and (i<11):
print('eat your can food')
SendReminderEmail(CanNumberActual)
#CanLastSent = datetime.datetime.now()
if j >= 30 and (j<31):
print('can food expired yo')
SendExpiryEmail(False,CanNumber=CanNumberActual)
#CanLastSent = datetime.datetime.now()
for i in RPiExpiry:
RPiExpiryDelta = datetime.datetime.now()-i.expiry
if (RPiExpiryDelta.total_seconds() >= 30) and (RPiExpiryDelta.total_seconds() < 31):
# Email
print('RPi Food Expired')
SendExpiryEmail(True,ImageFileName=i.imgPath)
#RPiLastSent = datetime.datetime.now()
# Parameters
# Initialize trackers
# Can
CanFirstOpenedRecord = CanFileToList(ReadFile("CanFirstOpened.txt"))
CanRecord = CanFileToList(ReadFile("CanLastOpened.txt"))
DeltaLastOpened,DeltaFirstOpened = CompareTime(CanFirstOpenedRecord,CanRecord)
# RPi
RPiExpiry = RPiFileToList(ReadFile("ExpirationTxtFile.txt"))
while True:
SendReminder(DeltaLastOpened,DeltaFirstOpened,RPiExpiry)
CanFirstOpenedRecord = CanFileToList(ReadFile("CanFirstOpened.txt"))
CanRecord = CanFileToList(ReadFile("CanLastOpened.txt"))
DeltaLastOpened,DeltaFirstOpened = CompareTime(CanFirstOpenedRecord,CanRecord)
RPiExpiry = RPiFileToList(ReadFile("ExpirationTxtFile.txt"))
time.sleep(1)
Feather Server on RPi
#!/usr/bin/env python2
import select
import socket
import datetime
def CanFileToList(FileOutput):
num = len(FileOutput)
ParsedFile=[]
for i in range(0,num-1):
ParsedFile.append(FileOutput[i][:-1])
return ParsedFile
def ReadFile(FileName):
# Read info file and update
CanFile = open(FileName,"r+")
CanFileOutput = CanFile.readlines()
CanFile.close()
return CanFileOutput
def WriteFile(FileName,Record):
# Write record into file
CanFile = open(FileName,"w")
ListToWrite = []
for i in Record:
ListToWrite.append(str(i)+"\n")
ListToWrite.append("END")
CanFile.writelines(ListToWrite)
CanFile.close()
def CompareTime(FirstOpened,LastOpened):
delta = []
for i,j in zip(FirstOpened,LastOpened):
FirstOpenedDT = datetime.datetime.strptime(i, '%Y-%m-%d %H:%M:%S.%f')
LastOpenedDT = datetime.datetime.strptime(j, '%Y-%m-%d %H:%M:%S.%f')
delta.append(LastOpenedDT-FirstOpenedDT)
return delta
def SendReminder(TimeDelta):
for i in TimeDelta:
if i.total_seconds() > 10:
print('eat your dang food')
return
# Parameters
numCans = 3
# Set up WiFi server listener
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('0.0.0.0', 8090 ))
s.listen(0)
#s.setblocking(0)
read_list = [s]
# Initialize trackers
CanFirstOpenedRecord = CanFileToList(ReadFile("CanFirstOpened.txt"))
CanRecord = CanFileToList(ReadFile("CanLastOpened.txt"))
#DeltaDT = CompareTime(CanFirstOpenedRecord,CanRecord)
#print(CanRecord)
#CanRecord[1] = str(datetime.datetime.now().time())
#print(CanRecord)
while True:
#print("Delta")
#print(DeltaDT)
#SendReminder(DeltaDT)
readable, writable, errored = select.select(read_list, [], [])
for i in readable:
if i is s:
client, addr = s.accept()
read_list.append(client)
else:
content = i.recv(32)
print(content)
lastContent = content
print("Closing connection")
i.close()
read_list.remove(i)
print("Updating can: "+lastContent[:1])
CanToUpdate = int(lastContent[:1])
if ("Reset" in lastContent):
CanFirstOpenedRecord[CanToUpdate-1] = str(datetime.datetime.now())
WriteFile("CanFirstOpened.txt",CanFirstOpenedRecord)
CanRecord[CanToUpdate-1] = str(datetime.datetime.now())
WriteFile("CanLastOpened.txt",CanRecord)
else:
CanRecord[CanToUpdate-1] = str(datetime.datetime.now())
WriteFile("CanLastOpened.txt",CanRecord)
print(CanFirstOpenedRecord)
print(CanRecord)
#DeltaDT = CompareTime(CanFirstOpenedRecord,CanRecord)
Sensor and Button Logic and WiFi Client on ESP32
#include
const char* ssid = "TheHottestSpot";
const char* password = "test12345";
const uint16_t port = 8090;
const char * host = "172.20.10.6";
bool signalDetected = false;
bool resetDetected = false;
uint8_t LED1pin = 13;
uint8_t SensorPin = 15;
const int PushButton = 33;
void IRAM_ATTR isr() { //this ISR definition mst go before void setup()
signalDetected = true;
}
void IRAM_ATTR isrButton() { //this ISR definition mst go before void setup()
Serial.println("ButtonPushed!");
resetDetected = true;
}
void SendOpenMessage(){
signalDetected = false;
WiFiClient client;
Serial.println("Before client connect");
if (!client.connect(host, port)) {
Serial.println("Connection to host failed");
delay(1000);
return;
}
Serial.println("Connected to server successful!");
client.print("1:Opened");
Serial.println("Disconnecting...");
client.stop();
}
void SendResetMessage(){
resetDetected = false;
WiFiClient client;
Serial.println("Sending reset");
if (!client.connect(host, port)) {
Serial.println("Connection to host failed");
delay(1000);
return;
}
Serial.println("Connected to server successful!");
client.print("1:Reset");
Serial.println("Disconnecting...");
client.stop();
}
void setup()
{
Serial.begin(115200);
pinMode(LED1pin, OUTPUT);
pinMode(SensorPin, INPUT_PULLUP);
pinMode(PushButton, INPUT);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.println("...");
}
Serial.print("WiFi connected with IP: ");
Serial.println(WiFi.localIP());
attachInterrupt(digitalPinToInterrupt(SensorPin), isr, RISING);
attachInterrupt(digitalPinToInterrupt(PushButton), isrButton, FALLING);
}
void loop()
{
// WiFiClient client;
// if (!client.connect(host, port)) {
//
// Serial.println("Connection to host failed");
//
// delay(1000);
// return;
// }
//
// Serial.println("Connected to server successful!");
//
// client.print("Hello from ESP32!");
//
// Serial.println("Disconnecting...");
// client.stop();
//
// delay(10000);
if (signalDetected){
SendOpenMessage();
}
if (resetDetected){
SendResetMessage();
}
}
RPi Picture Taking and OCR
#!/usr/bin/env python2
#Libraries to import
import RPi.GPIO as GPIO
import time
from picamera import PiCamera
from PIL import Image # python image library
import cv2
import numpy as np
import sys,pygame
import os
#import pytesseract
import datetime
os.putenv('SDL_VIDEODRIVER', 'fbcon') # Display on piTFT
os.putenv('SDL_FBDEV', '/dev/fb1') #
os.putenv('SDL_MOUSEDRV', 'TSLIB') # Track mouse clicks on piTFT
os.putenv('SDL_MOUSEDEV', '/dev/input/touchscreen')
pygame.init()
pygame.mouse.set_visible(False)
def blit_button_labels():
blit_text('Start',(280, 45))
blit_text('Take picture', (280, 110))
blit_text('Yes',(280, 170))
blit_text('No',(280, 230))
pygame.display.flip()
def blit_text(text,location):
text = my_font.render(text,True,white)
text_rect = text.get_rect(center=location)
screen.blit(text, text_rect)
pygame.display.flip()
# function to take picture
def take_picture(channel):
print('start of take picture callback')
global can_take_picture
global picture_count
global picture_item
global can_use_yn
global path_date
global path_item
print(can_take_picture)
camera.resolution = (2592,1944)
camera.framerate = 15
camera.brightness = 45
if can_take_picture and picture_item:
can_take_picture = False
screen.fill(black)
blit_text('Taking picture', (140,120))
blit_button_labels()
path_item = '/home/pi/FinalProject/TestImages/item_image'+str(picture_count)+'.jpg'
camera.capture(path_item)
#picture_count = picture_count + 1
display_picture(path_item)
time.sleep(2)
screen.fill(black)
blit_text('Retake picture?', (140,120))
blit_button_labels()
can_use_yn = True
if can_take_picture and picture_date:
can_take_picture = False
screen.fill(black)
blit_text('Taking picture', (140,120))
blit_button_labels()
path_date = '/home/pi/FinalProject/TestImages/expiration_image'+str(picture_count)+'.jpg'
camera.capture(path_date)
picture_count = picture_count + 1
display_picture(path_date)
time.sleep(2)
screen.fill(black)
blit_text('Retake picture?', (140,120))
blit_button_labels()
can_use_yn = True
#retake()
#if can_take_picture and picture_date:
def retake():
global answer
global can_take_picture
global picture_count
global picture_item
global picture_date
global image_processing_time
global can_use_yn
global manual_check
global manual_date
picture_count = picture_count - 1
if answer and picture_item:
can_take_picture = True
picture_count = picture_count + 1
screen.fill(black)
blit_text('Please take another picture of the item', (140, 120))
blit_button_labels()
can_use_yn = False
elif not(answer) and picture_item:
can_use_yn = False
picture_count = picture_count + 1
picture_item = False
picture_date = True
screen.fill(black)
blit_text("Please take a picture of the expiration date", (140, 120))
blit_button_labels()
can_take_picture = True
elif answer and picture_date:
can_take_picture = True
screen.fill(black)
blit_text('Please take another picture of the expiration date', (140, 120))
blit_button_labels()
can_use_yn = False
elif not(answer) and picture_date:
picture_date = False
image_processing_time = True
text = image_processing()
screen.fill(black)
blit_text('Is ' + text + ' the correct date?', (140, 120))
blit_button_labels()
can_use_yn = True
elif manual_check == True:
screen.fill(black)
blit_text('Is ' + manual_date + ' the correct date?', (140, 120))
blit_button_labels()
can_use_yn = True
def image_processing():
global path_date
global picture_count
# fill in for now\
img = cv2.imread(path_date, 0) #import expiration date picture in grayscale
cv2.imwrite(path_date[0:49] + str(picture_count) + 'gray.jpg', img)
return '12-12'
#text = pytesseract.image_to_string(Image.open(path))
#return text
def display_picture(path):
img_surface = pygame.image.load(path)
img_surface = pygame.transform.scale(img_surface, (320,240))
img_surface_rect = img_surface.get_rect()
#img_surface_rect_fit = img_surface_rect.fit(screen.get_rect())
screen.blit(img_surface, img_surface_rect)
pygame.display.flip()
def start_process(channel):
global start_of_process
global can_take_picture
global picture_item
if start_of_process:
screen.fill(black)
blit_text("Please take a picture of the item", (140, 120))
blit_button_labels()
start_of_process = False
can_take_picture = True
picture_item = True
def picture_taking_loop():
take_picture()
display_picture(path)
time.sleep(3)
screen.fill(black)
blit_text('Retake picture?', (140,120))
blit_button_labels()
def blit_button(color,text,location):
button_surface = my_font.render(' ',True, white)
button_circle = pygame.draw.circle(screen, color, location, 20,0)
button_text_surface = my_font.render(text,True,black)
button_text_rect = button_text_surface.get_rect(center=location)
screen.blit(button_surface, button_circle)
screen.blit(button_text_surface, button_text_rect)
pygame.display.flip()
def manual_date_func():
global manual_entry
global manual_count
global manual_date
global manual_check
manual_entry = True
screen.fill(black)
blit_button(white,'1',(80,60))
blit_button(white,'2',(160,60))
blit_button(white,'3',(240,60))
blit_button(white,'4',(80,110))
blit_button(white,'5',(160,110))
blit_button(white,'6',(240,110))
blit_button(white,'7',(80,160))
blit_button(white,'8',(160,160))
blit_button(white,'9',(240,160))
blit_button(white,'0',(160,210))
blit_text('Enter as MM-DD',(160, 20))
while manual_entry == True:
for event in pygame.event.get():
if(event.type is pygame.MOUSEBUTTONDOWN) and manual_entry == True:
pos = pygame.mouse.get_pos()
elif(event.type is pygame.MOUSEBUTTONUP) and manual_entry == True:
pos = pygame.mouse.get_pos()
x,y = pos
if x > 60 and x < 100 and y > 40 and y < 80:
print('button 1')
manual_count = manual_count + 1
manual_date = manual_date + '1'
elif x > 140 and x < 180 and y > 40 and y < 80:
print('button 2')
manual_count = manual_count + 1
manual_date = manual_date + '2'
elif x > 220 and x < 260 and y > 40 and y < 80:
print('button 3')
manual_count = manual_count + 1
manual_date = manual_date + '3'
elif x > 60 and x < 100 and y > 90 and y < 130:
print('button 4')
manual_count = manual_count + 1
manual_date = manual_date + '4'
elif x > 140 and x < 180 and y > 90 and y < 130:
print('button 5')
manual_count = manual_count + 1
manual_date = manual_date + '5'
elif x > 220 and x < 260 and y > 90 and y < 130:
print('button 6')
manual_count = manual_count + 1
manual_date = manual_date + '6'
elif x > 60 and x < 100 and y > 140 and y < 180:
print('button 7')
manual_count = manual_count + 1
manual_date = manual_date + '7'
elif x > 140 and x < 180 and y > 140 and y < 180:
print('button 8')
manual_count = manual_count + 1
manual_date = manual_date + '8'
elif x > 220 and x < 260 and y > 140 and y < 180:
print('button 9')
manual_count = manual_count + 1
manual_date = manual_date + '9'
elif x > 140 and x < 180 and y > 190 and y < 230:
print('button 0')
manual_count = manual_count + 1
manual_date = manual_date + '0'
if manual_count == 4:
print('exit manual entry mode')
manual_date = manual_date[0:2] + '-' + manual_date[2:4]
print(manual_date)
manual_entry = False
manual_check = True
retake()
def yes_callback(channel):
global answer
global can_use_yn
global picture_item
global picture_date
global image_processing_time
global manual_check
global start_of_process
global txt_file
global path_item
if can_use_yn == True:
print('pressed yes')
answer = True
if answer == True and picture_item == True:
retake()
elif answer == True and picture_date == True:
retake()
elif answer == True and image_processing_time == True:
image_processing_time = False
start_of_process = True
WriteFile = open(txt_file, 'w')
ListToWrite = []
ListToWrite.append(str(datetime.datetime.now()) + '\n')
ListToWrite.append(str(path_item))
WriteFile.writelines(ListToWrite)
WriteFile.close()
screen.fill(black)
blit_text('Thanks! All set :)', (140, 120))
blit_button_labels()
elif answer == True and manual_check == True:
manual_check = False
start_of_process = True
screen.fill(black)
blit_text('Thanks! All set :)', (140, 120))
blit_button_labels()
def no_callback(channel):
global answer
global can_use_yn
global picture_item
global picture_date
global image_processing_time
global manual_check
global manual_date
if can_use_yn == True:
print('pressed no')
answer = False
if answer == False and picture_item == True:
retake()
elif answer == False and picture_date == True:
retake()
elif answer == False and image_processing_time == True:
image_processing_time = False
screen.fill(black)
blit_text('Please manually type the date', (140, 120))
blit_button_labels()
time.sleep(1.1)
image_processing_time = False
manual_date_func()
elif answer == False and manual_check == True:
manual_check = False
manual_date = ''
screen.fill(black)
blit_text('Please manually type the date again', (140, 120))
blit_button_labels()
time.sleep(1.1)
manual_date_func()
# Initalize necessary GPIO pins
GPIO.setmode(GPIO.BCM)
GPIO.setup(17, GPIO.IN, pull_up_down=GPIO.PUD_UP)
GPIO.setup(22, GPIO.IN, pull_up_down=GPIO.PUD_UP) # button on RPi to take picture
GPIO.setup(23, GPIO.IN, pull_up_down=GPIO.PUD_UP)
GPIO.setup(27, GPIO.IN, pull_up_down=GPIO.PUD_UP)
# set up camera
camera = PiCamera()
# Initialize global variables
global picture_count
global path
global start_of_process
global can_take_picture
global answer
global picture_item
global picture_date
global image_processing_time
global can_use_yn
global manual_entry
global manual_count
global manual_date
global manual_check
global path_date
global txt_file
# Define global variables
picture_count = 1
start_of_process = True
can_take_picture = False
picture_item = False
picture_date = False
image_processing_time = False
can_use_yn = False
manual_entry = False
manual_count = 0
manual_date = ''
manual_check = False
path_date = ''
txt_file = '/home/pi/FinalProject/ExpirationTxtFile.txt'
size = width, height = 320, 240
black = 0,0,0
white = 255, 255, 255
my_font= pygame.font.Font(None, 16)
screen = pygame.display.set_mode(size)
screen.fill(black)
blit_button_labels()
blit_text('Please press start to begin!', (140,120))
# function to process picture
#def process_picture():
GPIO.add_event_detect(22, GPIO.FALLING, callback = take_picture, bouncetime = 300)
GPIO.add_event_detect(17, GPIO.FALLING, callback = start_process, bouncetime = 300)
GPIO.add_event_detect(23, GPIO.FALLING, callback = yes_callback, bouncetime = 300)
GPIO.add_event_detect(27, GPIO.FALLING, callback = no_callback, bouncetime = 300)
# MAIN LOOP
print('starting')
a = True
while a:
time.sleep(30)
a = False
Bash script to start the whole program
#!/usr/bin/env bash
trap "exit" INT TERM ERR
trap "kill 0" EXIT
python FeatherServerMulti.py -e &
python ExpiryMonitor.py -e &
wait